{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[View the runnable example on GitHub](https://github.com/intel-analytics/BigDL/tree/main/python/nano/tutorial/notebook/inference/tensorflow/accelerate_tensorflow_inference_openvino.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Accelerate TensorFlow Inference using OpenVINO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can use ``InferenceOptimizer.trace(..., accelerator='openvino')`` API to enable the OpenVINO acceleration for TensorFlow inference. It only takes a few lines."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nbsphinx": "hidden"
      },
      "source": [
        "To apply OpenVINO acceleration, the following dependencies need to be installed first:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "nbsphinx": "hidden"
      },
      "outputs": [],
      "source": [
        "!pip install --pre --upgrade bigdl-nano[tensorflow,inference] # install the nightly-built version\n",
        "!source bigdl-nano-init # set environment variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nbsphinx": "hidden"
      },
      "source": [
        "> 📝 **Note**\n",
        ">\n",
        ">  We recommend to run the commands above, especially `source bigdl-nano-init` before jupyter kernel is started, or some of the optimizations may not take effect."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's take an [EfficientNetB0 model](https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet/EfficientNetB0) pretrained on ImageNet dataset as an example. First, we load the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from tensorflow.keras.applications import EfficientNetB0\n",
        "\n",
        "model = EfficientNetB0(weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To enable OpenVINO acceleration for your TensorFlow inference, **the only change you need to made is to import BigDL-Nano** `InferenceOptimizer`**, and trace your TensorFlow model to convert it into an OpenVINO accelerated module for inference**:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from bigdl.nano.tf.keras import InferenceOptimizer\n",
        "\n",
        "ov_model = InferenceOptimizer.trace(model,\n",
        "                                    accelerator=\"openvino\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> 📝 **Note**\n",
        ">\n",
        "> Note that when you have a custom model (e.g. inherited from `tf.keras.Model`), parameter `input_spec`, which should be a (list or tuple of) `tf.TensorSpec`, is required for the `trace` function to let OpenVINO accelerator know the shape of the model input.\n",
        ">\n",
        "> Please refer to [API documentation](https://bigdl.readthedocs.io/en/latest/doc/PythonAPI/Nano/tensorflow.html#bigdl.nano.tf.keras.InferenceOptimizer.trace) for more information on `InferenceOptimizer.trace`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You could then do the normal inference steps with the model optimized by OpenVINO:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "x = tf.random.normal(shape=(2, 224, 224, 3))\n",
        "# use the optimized model here\n",
        "y_hat = ov_model(x)\n",
        "predictions = tf.argmax(y_hat, axis=1)\n",
        "print(predictions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> 📚 **Related Readings**\n",
        "> \n",
        "> - [How to install BigDL-Nano](https://bigdl.readthedocs.io/en/latest/doc/Nano/Overview/install.html)\n",
        "> - [How to install BigDL-Nano in Google Colab](https://bigdl.readthedocs.io/en/latest/doc/Nano/Howto/Install/install_in_colab.html)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "nano-tf",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.16 (default, Jan 17 2023, 23:13:24) \n[GCC 11.2.0]"
    },
    "nteract": {
      "version": "0.28.0"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "402532f56d486e9f832908f31130bbdf12bd8cb099dfb226783aa2c6b1479100"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}
